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The RICIS Concept 


The University of Houston-Clear Lake established the Research Institute for 
Computing and Information Systems (RICIS) in 1 986 to encourage the NASA 
Johnson Space Center (JSC) and local industry to actively support research 
in the computing and information sciences. As part of this endeavor, UHCL 
proposed a partnership with JSC to jointly define and manage an integrated 
program of research in advanced data processing technology needed for JSC’s 
main missions, including administrative, engineering and science responsi- 
bilities. JSC agreed and entered into a continuing cooperative agreement 
with UHCL beginning in May 1986, to jointly plan and execute such research 
through RICIS. Additionally, under Cooperative Agreement NCC 9- 16, 
computing and educational facilities are shared by the two institutions to 
conduct the research. 

The UHCL/RICIS mission is to conduct, coordinate, and disseminate research 
and professional level education in computing and information systems to 
serve the needs of the government, industry, community and academia. 
RICIS combines resources of UHCL and its gateway affiliates to research and 
develop materials, prototypes and publications on topics of mutual interest 
to its sponsors and researchers. Within UHCL, the mission is being 
implemented through interdisciplinary involvement of faculty and students 
from each of the four schools: Business and Public Administration, Educa- 
tion, Human Sciences and Humanities, and Natural and Applied Sciences. 
RICIS also collaborates with indu stry in a companion program. This program 
is focused on serving the research and advanced development needs of 
industry. 

Moreover* UHCL established relationships with other universities and re- 
search organizations, having common research interests, to provide addi- 
tional sources of expertise to conduct needed research. For example, UHCL 
has entered into a special partnership with Texas A&M University to help 
oversee RICIS research and education programs, while other research 
organizations are Involved via the “gateway" concept 

A major role of RICIS then is to find the best match of sponsors, researchers 
and research objectives to advance knowledge in the computing and informa- 
tion sciences. RICIS, working jointly with its sponsors, advises on research 
needs, recommends principals for conducting the research, provides tech- 
nical and administrative support to coordinate the research and integrates 
technical results into the goals of UHCL, NASA/JSC and industry. 




Preface 


This research was conducted under auspices of the Research Institute for 
Computing and Information Systems by Dr. Pamela K. Fink of the Southwest 
Research Institute. Dr. Glenn B. Freedman served as RICIS research coordinator. 

Funding has been provided by the Mission Operations Directorate, 
NASA/JSC through Cooperative Agreement NCC 9-16 between the NASA 
Johnson Space Center and the University of Houston-Clear Lake. The NASA 
technical monitor for this activity was Barbara N. Pearson, of the 
Systems/Elements Office, Space Station Training Office, Mission Operations 
Directorate, NASA/JSC. 

The views and conclusions contained in this report are those of the author 
and should not be interpreted as representative of the official policies, either 
express or implied, of NASA or the United States Government. 
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September 11, 1990 


Intelligent Systems Evaluation 
Training Systems Division 
AFHRL/ID 

Brooks AFB, TX 78235 
Attention: Kurt Steuck, Ph.D. 

Subject: Monthly Progress Report from August 20 through August 31, 1990; 

Research Activity No. ET.26; NASA Cooperative Agreement NCC9-16; 
Southwest Research Institute (SwRI) Project No. 05-3744 

Gentlemen: 

Southwest Research Institute was put under contract to perform the proposed 
work in developing a knowledge engineering taxonomy on August 20, 1990. Dr. Pam 
Fink and Dr. Tandy Herren met with Kurt Steuck and Capt. B.J. Jones of HRL on 
August 28, 1990 to discuss the goals, direction, and schedule of the project. 
We also began generating a list of potential problem solving tasks that could 
be considered for closer examination during the course of the project. Our first 
task is to identify a set of specific problem solving tasks, along with at least 
one human source of expertise in each area. A preliminary list has been 

generated. In order to support selection of an appropriate and representative 
set of problem solving tasks , we are examining existing problem solving and 
learning taxonomies. These taxonomies are being used to generate a large matrix 
of tasks along one axis and attributes of tasks, such as reasoning methods, 
sources of data, problem solving environment, etc., along the other. The 
selected set of problem solving tasks should cover a large portion of this 
matrix. 

During the next reporting period we will complete the selection of a set 
of problem solving tasks. We will also begin the design of the interview 
sessions to be performed with the experts. Issues to be resolved include: 

interviewing one domain expert in each area twice or two domain experts 
in each area once, 

performing all interviewing for a given problem solving task at one time 
or perform preliminary interviews with each expert first and then return 
for a second interview, and 

what knowledge engineering techniques should be used in each case. 
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Expenditures for this reporting period totaled $1,862.23, leaving a balance 
of $64,046.77 of the original $65,909 contract amount. 


Approved: 



Vice President 
Automation & Data 
Division 


Systems 


Sincerely, 

Pamela K. Fink, Ph.D. 
Manager 

Artificial Intelligence 
Section 


RBC/PKF/em 

Enclosures 

cc: Nancy Bell, RICIS 

Bill Bayliss, SwRI 
Tandy Herren, SwRI 
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October 10, 1990 


Intelligent Systems Evaluation 
Training Systems Division 
AFHRL/ID 

Brooks AFB, TX 78235 
Attention: Kurt Steuck, Ph.D. 

Subject: Monthly Progress Report from September 1 through September 28, 1990; 

Research Activity No. ET.26; NASA Cooperative Agreement NCC9-16; 

Southwest Research Institute (SwRI) Project No. 05-3744 

Gentlemen: 

The emphasis during this reporting period has been on the generation of 
three lists: 1) a list of possible problem solving tasks/jobs to be considered, 
2 ) a list of task characteristics to be utilized in an initial attempt to 
discriminate between different problem solving tasks, and 3) a list of knowledge 
acquisition techniques, both automated and non- automated. The list of possible 
problem solving tasks consists of approximately 35 jobs ranging from equipment 
diagnosis to facility management to chemical analysis. The list attempts to 
capture as many different areas of problem solving that are of interest to the 
Air Force and NASA as possible. The list of problem solving task characteristics 
consists of approximately 16 major categories of issues, including reasoning 
techniques used, inputs available, motor processes required, and the environment 
in which the task is performed. Each of these major areas includes a number of 
sub-characteristics. For example, reasoning techniques include statistical, 
spacial, temporal, analogical, case-based, modelling, mathematical, deductive, 
and inductive. The list of knowledge acquisition techniques includes non- 
aucomated, or traditional, techniques such as written materials , verbal reports, 
and psychometric techniques, as well as automated techniques where approximately 
40 tools categorized as either induction-based or interview-based are enumerated. 

Once all of these lists were generated, a meeting was held on September 
26 at HRL between Pam Fink, Tandy Herren, Kurt Steuck, and B.J. Jones concerning 
the completeness of the lists and the planned approach for utilizing them. A 
number of additional problem solving tasks/jobs were added, and the Issue of 
communication was added to the task characteristics list. Based on the feedback 
from this meeting, a large matrix will be generated that crosses problem solving 
tasks/jobs with problem solving task characteristics. Check marks will be placed 
in the appropriate boxes indicating which characteristics are relevant to which 
tasks/jobs. A copy of this matrix is due to be sent to Dr. Steuck and Capt. 
Jones around October 3. This matrix will be reviewed and modified as appropriate 
and will then be used to support selection of the 10-12 problem solving 
tasks/jobs for further study. A meeting is set for October 12. 
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SCHEDULE FOR RESEARCH IN DEVELOPMENT 
OF A KNOWLEDGE ENGINEERING TAXONOMY 


Months from Date of Award 


1990 1991 


TASK | Aug Sep Oct Nov Dec Jan Feb Mar Apr May 


1 | D(9/28) 

2 | (2/28) D(5/10) 

3 I (3/30) 

4 I (4/26) 

5 I (3/30) 

6 | (4/26) 

7 | CDDDDDD D D(5/19) 


List of Tasks : 

Task 1 - Identification of a Set of Specific Problem Solving Tasks 

Task 2 - Definition of Identified Problem Solving Tasks 

Task 3 - Analysis of Identified Problem Solving Tasks 

Task 4 - Development of a Proposed Taxonomy of Problem Solving Tasks 

Task 5 - Analysis of Knowledge Engineering Techniques Used in Task 2 

Task 6 - Matching of Knowledge Engineering Techniques to Problem Solving Tasks 

Task 7 - Reporting 


afhrl/id 

October 10, 1990 
Page Two 


Based on the original schedule, we were to have selected the 10-12 
tasks/jobs by September 28 (Task 1). However, in order to do a better job of 
selecting a set of representative tasks, we have placed more emphasis up front 
on defining and analyzing the problem solving tasks (Tasks 2&3) . Based on the 
meeting set for October 12, we will make a selection of the 10-12 tasks and begin 
scheduling interviews. The schedule has been modified to reflect this change. 
We hope that having some knowledge up front of the various tasks, in terms of 
the type of problem solving that is performed, will allow for a more 
representative selection of tasks for further study. We also believe that such 
knowledge will be helpful in designing the interviews to be performed with the 
domain experts. Currently, we plan to perform interviews in two stages. The 
first stage will involve a high-level interview to better define each task in 
terms of the list of problem solving characteristics and to refine the list of 
characteristics. The second stage will then attempt to match knowledge 
acquisition techniques based on the characteristics and assess the results. 

Expenditures for this reporting period totaled $3,317.95, leaving a balance 
of $60,728.82 of the original $65,909 contract amount. 

Sincerely, 

Pamela K. Fink, Ph.D. 

Manager 

Artificial Intelligence 
Section 


Approved: 



/ Richard B. Curtin 
Vice President 
Automation & Data Systems 
Division 

RBC/PKF/em 

Enclosure 

cc: Nancy Bell, RICIS 

Bill Bayliss, SwRI 
Tandy Herren, SwRI 



SCHEDULE FOR RESEARCH IN DEVELOPMENT 
OF A KNOWLEDGE ENGINEERING TAXONOMY 


Monchs from DaCe of Award 
1990 1991 

TASK | Aug Sep Oct Nov Dec Jan Feb Mar Apr May 

1 | D(10/18) 

2 | (2/28) D(5/10) 

3 | * (3/30) 

4 | (4/26) 

5 | (3/30) 

6 | (4/26) 

7 | CCDDDDD D D(5/19) 

List of Tasks: 

Task 1 - Identification of a Set of Specific Problem Solving Tasks 

Task 2 - Definition of Identified Problem Solving Tasks 

Task 3 - Analysis of Identified Problem Solving Tasks 

Task 4 - Development of a Proposed Taxonomy of Problem Solving Tasks 

Task 5 - Analysis of Knowledge Engineering Techniques Used in Task 2 

Task 6 - Matching of Knowledge Engineering Techniques to Problem Solving Tasks 

Task 7 - Reporting 
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November 9 , 1990 


Intelligent Systems Evaluation 
Training Systems Division 
AFHRL/ID 

Brooks AFB t TX 78235 
Attention; Kurt Steuck, Ph.D. 

Subject: Monthly Progress Report from September 29 through October 26, 

1990; Research Activity No. ET.26; NASA Cooperative Agreement 
NCC9-16; Southwest Research Institute (SwRI) Project No, 05-3744 

Gentlemen : 

A copy of the matrix crossing problem characteristics with selected tasks 
was mailed to Brooks for review on October 2, 1990. This matrix served as our 
basis for discussion in a meeting held between Dr. Kurt Steuck, Capt. B. J. 
Jones, Dr. Tandy Herren, and Dr. Pam Fink on October 12. Based on this meeting, 
a set of tasks were selected and responsibilities assigned concerning who was to 
provide a contact. The list of tasks initially analyzed, a list of distinctive 
task types, and the list of tasks selected for further analysis are attached. 
Some experts have been identified and some initial interviews have taken place. 
These are indicated by the names and dates listed with the specific tasks . These 
initial interviews have been based on a questionnaire designed to help us better 
define the values that should be placed next to particular task characteristics 
for the given task. A copy of the questionnaire is attached. When we interview, 
we do not go through the entire questionnaire but use it as a guide to ensure 
that we get all of the information we need. Questions marked with a "**" 
indicate we believe these are key questions. 

During the next reporting period, we plan on setting-up and carrying-out the 
initial interviews not yet performed. Based on the initial interview, we are re- 
doing the problem characteristics for each selected task and using a scale of 0- 
4, 0 indicating not important and 4 indicating key importance to the particular 

cask. 



SAN ANTONIO, TEXAS 

HOUSTON. TEXAS • DETROIT. MICHIGAN 


WASHINGTON, OC 



AFHRL/ID 

November 9, 1990 
Page Two 



cc: Nancy Bell, RICIS 

Bill Bayliss, SwRI 
Tandy Her ten, SwRI 


SCHEDULE FOR RESEARCH IN DEVELOPMENT 
OF A KNOWLEDGE ENGINEERING TAXONOMY 


Months from Date of Award 
1990 1991 


TASK 
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List of Tasks : 

Task 1 - Identification of a Set of Specific Problem Solving Tasks 

Task 2 - Definition of Identified Problem Solving Tasks 

Task 3 - Analysis of Identified Problem Solving Tasks 

Task 4 - Development of a Proposed Taxonomy of Problem Solving Tasks 

Task 5 - Analysis of Knowledge Engineering Techniques Used' in Task 2 

Task 6 - Matching of Knowledge Engineering Techniques to Problem Solving Tasks 

Task 7 - Reporting 



TASK/ JOB LIST FOR 
KNOWLEDGE ENGINEERING TAXONOMY 


The following is a list of the tasks/jobs initially identified for 
preliminary analysis: 

1. maintain systems/diagnostics 

2. repair systems/making the fix 

3. communications monitoring/analysis 

4. photo analysis 

5. training pilots 

6. training astronauts 

7. training air crews (groups to work together) 

8. training instructors 

9. air traffic control 

10. console operations 

11. weather forecasting/modelling 

12. program management 

13. facilities management 

14. cargo loading 

15. operating a piece of equipment 

16. software design 

17. software maintenance/debugging 

18. form fill-out 

19. training reading 

20. training foreign language 

21. oil accident investigation 

22. prospecting 

23. power grid management 


Page 2 


24. computer configuration 

25. chemical analysis 

26. personnel management 

27. customer service 

28. sales 

29. weapons director/air intercept 

30. use RMS/other remote manipulator 

31. leadership training 

32. surgery 

33. medical diagnosis 

34. accounting 

35. marketing/advertising 

36. knowledge engineering 

37. scientific protocol design 

Based on the preliminary analysis, a set of distinctive attributes 
about tasks became apparent. The set of tasks that should be analyzed 
further should include at least one representative of each of the 
following: 

1. diagnostic task 

2. training task 

3. high performance task 

4. form fill-out 

5. people-oriented (soft) task 

6. design task 

7. planning task 

8. monitoring with a time factor 

9. perceptually-oriented task 

10. bin-packing/np-complete task 
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11. numerical task 

12. data intensive (no real time factor), ie. acquire and present 

The following is the list of selected tasks/jobs for which an 
initial interview will be performed (dates indicate date of initial 
interview) : 

1. (diagnositcs) medical diagnosis - HRL 

2. (diagnostics) equipment diagnosis - ?? 

3. (training) training pilots - HRL/Randolph 

4. (training) training a foreign language - HRL 

5. (high performance) flight controller console operations - Matt Barry, 
NASA/ JSC - early Nov. 

6. (high performance/KR) surgery - HRL/BAMC 

7. form fill-out - Bill Bayliss, SwRI Contracts - Oct. 26, 1990 

8. (people-oriented) personnel management - HRL/CBPO 

9. (design) sofware design - Susan Crumrine, SvRI - ?? 

10. (planning) acquisition program management - Xavier Pena, Kelly AFB - 

?? 

11- (monitoring/time) air traffic control - David Settle, FAA - Nov. 1, 
1990 

12. (perceptual) ?? 

13. (bin-packing) cargo loading - HRL/Kelly 

14. (numerical) accounting - Linda Boehme, SwRI CPA - ?? 

15. (data intensive/no time) ?? 

16. (??) communications analysis - HRL/ESC 

17. (7?) chemical analysis - HRL/OEHL drug testing 


} >nen you are def ining/identifving the problem, do you spend aloe 
' f time looking up information or chinking about different aspects 
H the probLem or do you perform the cask mostly automatically? 
."hac types of interactions, if any, are required in terms of 
iocuments , data systems, or other people when performing problem 
:ef inition? 


-.11- '/hat types of information come to mind when you do the task in 
the sense of inputs into the problem? 

- tacts 

* principles /rules/ laws 

- procedures 

- similar problems/analogies 

* past problems 

■•Hi JO you interact with other people while you're defining the problem? 

If so. how? 

1 o you : 

- Advise 

- answer 

- direct 

- indicate 

- inform 

- instruct 

- request 

- transmit 

- supervise 

■--- lo you think body language such as gestures and facial expresssions 
■’»re important in communicating at this time? 

*.-.12/ Is there alot of uncertainty in the problem -- maybe even to the 

extent that it's hard to know if there is a problem? How do you go 
about identifying/defining the problem? 

.14) Do the problem characteristics change frequently, maybe even while 
you're solving it? If so, what changes -- data, rules by which you 
solve the problem? What drives the change - e.g., time intervals? 

--.15) Do you have to solve the problem quickly or are there other 
constraints on the problem? e.g., Is time a factor - what is 
"real- time" in this situation -- seconds, minutes, hours? How 
flexible are any constraints on the problem? 

'••A16) Do you use solutions to past problems to help you solve the current 
problem? How similar do they tend to be and how do you modify them? 

A17) How do you decide what past problem to use?. Do you use a specific 
problem or some general version? 


A18) Do you try to draw parallels to similar but not identical problems, 


'.carview Questions 
r Lcial Discussion: 

There are no right answers or good answers to these questions. Some of the questions 
uv be phrased so that you believe one or another response is appropriate or somehow 
otter than another, "or example, it may sound good to say you plan up front all your 
.c cions in solving a problem or doing the job even if some of those actions occur somewhat 
uccmatically . Please do not fall into this bias. For this example, it is good to plan but 
t is also good not to plan too much and to have some actions chat occur automatically. 
Da want to assess all aspects of your job with as much fidelity as possible. What results 
: com these interviews will help us to better understand the unique aspects of your work 
na how we could learn more about it in the future so we can design computer systems that 
.zip you perform your job more easily. 

Also, we want to assure you that under no circumstances will your answers go to your 
ipervisor or anyone else outside of this room. You are responding to this questionnaire 
omo lately anonomvous ly . Because of this, we want you to provide the operational answers 
: rr.ese questions, not the official ones. We wane to know what you really do on your job. 

Understanding the Problem 

‘ A L : Tell ns a little bit about your field, yourself, and how you 
got into this field. 

•A l) Yhac initiates the problem solving process? 

'That types of inputs to the problem do you have? How do you 
recognize the problem? 

- Do you 

- see something 

- hear something 

- talk to someone 

- feel something 

- read something -- book, test equipment, output 

computer display 

- detect it based on past data 

■‘AM Does solving the problem require alot of data? If so, about what 
and how do you acquire the data? Do you try to group data 
together into categories” that have some unifying principle? 

A5) Is the data you work with primarily numerical or is it more 
qualitative? 

\o) What assumptions can be made that would simplify the (tablets 
solving task? What is the effect when these assumptions are 
removed? a- - ; 

■\ 7) What could happen to complicate the task? , 


-.8) .Are there manuals or desk procedures that document/direct what you 

do? 


ORIGINAL PAGE IS 
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see how chose problems were solved, and use chat information in che 
:urrenc problem? 

■•■"hat equipment do you use while defining che problem? 

■--0) How many alternative answers are there for a problem? Do you have 
-.o check for many as you solve ic? Is one answer obviously better 
than another? How can you tell? Must you find the best answer or 
is a certain level of answer good enough? 

'■Ail) How would you describe the output of your task? Do you generate a 
document, a decision, a recommendation, an action, a piece of 
equipment, an interaction with someone, etc.? 

.>.2; ran you list all possible solutions to your problem at the start? 
if so. approximately how many are there? 


revising the Plan 

-323) When you think about a problem do you 
a) visualize it in your mind 

h) formalize ic in a set of equations and work It out mathematically 
c) explain it and solve ic in English 

i Do you calculate or look at statistics to help you solve the problem? 

22 5) Do you order elements of the problem in time or do you have to think 
about a time element during the solution? (i.e., such as data or 
input changing over time) . 

226 i Do you generate a model to help you solve the problem, such as 
a mathematical model, a physical model, or some other symbolic 
representation of the problem? Do you "simulate 11 the behavior of 
a device in your head. 

227) When you plan che job, do you spend alot of time looking up 
information or thinking about different aspects of the problem 
or do you perform the task mostly automatically? What types of 
interactions are required in terms of documents, data systems, 
or other people? 

228) Do you break the problem Into parts? Are there sequential tasks 
you perform or logically distinct subcomponents of the problem? 

If so, what are they? 

329) When you plan the task, do you have to do one thing after another, 
such that one set of conditions causes you to respond with a whole 
series of actions? Do you wait after each one to see the results 
before you go any further? 


330) Do you interact with other people while planning the task? If so. 


ORIGINAL PAGE IS 
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how? 


jo you: 

- advise 

- answer 

- direct 

• indicate 

- inform 

- instruct 

- request 

- transmit 

- supervise 

- body language 

331 j What type of formal education do you have? Both in general as well 
as specific to your job. What role does experience play? Do 
you have an apprenticeship? How do you become qualified for what you 

do? 

53 2) Oo you think personal factors such as age, gender, height, or weight 
influence yours or others ability to do the task? 

Ill) Do you think you have to have a certain attitude or personality to 
work well on this job? 

334) Do you generate solutions and then test them to see if they work? 

335) Do you chink of past solutions and then modify them to suit the 
current problem? 

536) Do you look at data and find the problem or a method, then look at 
more data, etc. Or do you have a goal and collect data to meet that 
goal? 

33 7, When you solve the problem do you have a goal and take incremental 
steps in an attempt to step by step get to that goal? 

338) Are there set procedures or formulas you use for your task? 

339) Are the procedures you use to solve the problem known? 


340) Do you gather data and then present it by writing about it or 
discussing it with others? 

•’-*B4l) Do you b rea k your problem into sub-parts and then s ol ve th ose? 

After you solve the sub-parts do you have to reassemble them into 
an overall solution? __ 

342) Do you ever go back and return to the specifications or original 
definition of the problem? 

343) Do you generate solutions that explain part of the problem, and then 
determine what part is solved and try to generate a solution to solve 


any remaining portions of the problem? 

344) During your task do you have to 

- calculate 

- code 

- computerize 

- interpolate 

- itemize 

- tabulate 

- translate 

- estimate 

- compute 

345) Do you choose between alternatives? Do you compare alternatives? 

346) Do you integrate portions of the problem in the final solution? 

347) Do you have to do alot of planning? Plan the task -- or create a 

plan? 

348) Do you categorize parts of the problem together? Or categorize data 
in order to solve the problem? 

349) Are there general principles that you know to be true in your 
field? Do you use those to help determine what's going on in a 
specific situation? 

••• B50) If you make a wrong choice, is it easy to undo and if it is, how do 
you undo it? 

351) -hat equipment do you use when planning the problem solution? 


'■) Carrying Out the Problem 

•*C52) What are the major steps you perform? 

753) Where is your problem solving generally performed? Describe your 
work environment. Do you work at a desk with paper and pencil, at 
a console or in front of a piece of equipment, walking around and 
calking to people, or flying an aircraft? 

Is there: 

- acceleration 

- confinement 

- isolation 

- contaminants - what are they? 

- electricity 

- lighting 

• magnetism 

• noise 


354 ) How demanding is the task physically and psychologically? 

- fatigue 

- mental strain 

- stress 

- physical strain 

- precision 

- amount of sleep 

- work schedule 

- have to pay attention all the time 

C55) Do you have to pay close attention to the task for long periods of 
time? 

356 ) What types of physical movements do you make when you do the job? 

(we will assess motor processes and perceptual requirements 
from the demonstration) 

357) Do you interact with other people during the actual execution of 
the task? How? 

Do you: 

- advise 

- answer 

- direct 

- indicate 

- inform 

- instruct 

- request 

- transmit 

- supervise 

- body language 

■358) '-Then you do the job, do you spend alot of time looking up information 
or thinking about different aspects of the problem or do you 
perform the task mostly automatically? What types of interactions are 
required in terms of documents, data systems, or other people during 
task execution? 

C59) Do you monitor some set of Indicators as the task progresses? 

C60) Do you have to interpret Information you receive maybe forming new 
indicators? 

C61) What type of equipment is involved in actually performing the task? 

C62) When you perform the job, do you have to do one thing after another, 
such that one set of conditions causes you to respond with a whole 
series of actions? Do you wait after each one to see the results 
before you go any further? 

••*C63) What is the hardest/easiest thing you do? What is the most 


important 
3) Looking Back 


-■*064) How many alternative answers do you usually come up with? Do you 
have to check for many as you solve it? Is one answer obviously 
better than another? How can you tell? Must you find the best answer 
or is a certain level of answer good enough? 

36 5 ) Do you interact with other people in verifying the task? How? 

Do you: 

- advise 

- answer 

- direct 

- indicate 

- inform 

- instruct 

- request 

- transmit 

- supervise 

- body language 

D66 ) Vhen you evaluate your solution, do you spend alot of time looking 
up information or thinking about different aspects of the problem? 

367 ) Can the quality of a solution be characterized as better or worse 
rather than acceptable/not acceptable? 
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December 10. 1990 


(:."e II igent Systems Evaluation 
Training Systems Division 
ArHRL/ID 

3 rooks AFB. TX "8235 
.-.ttention: Kurt Steuck, Ph.D. 

v.ibiect: Monthly Progress Report from October 27 through November 23, 

1990; Research Activity No. ET.26; NASA Cooperative Agreement 
NCC9-16; Southwest Research Institute (SwRI) Project No. 05-3744 

■i-ntiemen : 

During this reporting period, we have set-up and performed some of the 
initial interviews in the areas agreed upon. We have scheduled meetings in 
'..inning acquisition program management) , monitoring with a time constraint (air 
-rut fie control), and high performance (shuttle flight control). We have 
rerformed the air traffic control interview. Based on the initial interview in 
,-;,ch, we are re-doing the problem characteristics for each selected task and 
using a scale of 0-4, 0 indicating not important and 4 indicating key importance 
to the particular task. 

During the next reporting period, we will continue to set-up and perform 
initial interviews. An updated copy of the task list is attached, indicating 
.-at has been done and dates, as well as responsibilities for making initial 
contacts . 


Expenditures for this reporting period totaled $3,236.23. leaving a balance 
l ;■ 5 2 . 1 1 1 . 1 2 of the original $65,909 contract amount. 


/ ;,p roved : 



/ Richard B. Curtin 
'/ice President 
.Automation & Data Systems 
Division 


S incerelv , 


/ £Ly*s1j3l 


Pamela K. Fink, Ph.D. 
Manager 

Artificial Intelligence 
Section 


RBC/PKF/em 
Enc losure 

oc; Nancy Bell, RICIS 
Bill Bayliss, SwRI 
w Tandv Herren, SwRI 
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SCHEDULE FOR RESEARCH IN DEVELOPMENT 
OF A KNOWLEDGE ENGINEERING TAXONOMY 


Months from Date of Award 


1990 199 L 


TASK 

1 Aug 

Sep 

Oct Nov 

Dec 

Jan Feb Mar Apr 

May 


1 


- -rn n/i fn 




L 

2 

3 

1 

1 




(2/28) 

( 1 /lO'i 

D(5/10) 

1 




f 4 /? M 

......... 


4 

5 

1 

1 

1 




(3/30) 

( L /'?&') 


0 

7 

1 

1 

C 

c c 

C 

D D D D- - 

D(5/19) 


Ls z of Tasks : 

isk 1 - Identification of a Set of Specific Problem Solving Tasks 
;sk 2 - Definition of Identified Problem Solving Tasks 
ask 3 - Analysis of Identified Problem Solving Tasks 
tsk 4 - Development of a Proposed Taxonomy of Problem Solving Tasks 
,sk 5 - Analysis of Knowledge Engineering Techniques Used in Task 2 
-sk 6 - Matching of Knowledge Engineering Techniques to Problem Solving Tasks 


.sk 7 - Reporting 



TASK/ JOB LIST FOR 
KNOWLEDGE ENGINEERING TAXONOMY 


The following is a list of the tasks/jobs initially identified for 
preliminary analysis: 

1. maintain systems/diagnostics 

2. repair systems/making the fix 

3. communications monitoring/analysis 

4. photo analysis 

5. training pilots 

6. training astronauts 

7. training air crews (groups to work together) 

8. training instructors 

9. air traffic control 

10. console operations 

11. weather forecasting/modelling 

12. program management 

13. facilities management 

14. cargo loading 

15. operating a piece of equipment 

16. software design 

17. software maintenance/debugging 

18. form fill-out 

19. training reading 

20. training foreign language 

21. oil accident investigation 

22. prospecting 

23. power grid management 
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24. computer configuration 

25. chemical analysis 

26. personnel management 

27. customer service 

28. sales 

29. weapons director/air intercept 

30. use RMS/other remote manipulator 

31. leadership training 

32. surgery 

33. medical diagnosis 

34. accounting 

35. marketing/advertising 

36. knowledge engineering 

37. scientific protocol design 

Based on the preliminary analysis, a set of distinctive attributes 
about tasks became apparent. The set of tasks that should be analyzed 
further should include at least one representative of each of the 
following: 


1 . 

diagnostic task 


2. 

training task 


3. 

high performance task 


4. 

form fill-out 


5. 

people-oriented (soft) 

task 

6. 

design task 


7. 

planning task 


8. 

monitoring with a time 

factor 

9. 

perceptually-oriented task 

10. 

bin- packing/ np-complete 

task 
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11. numerical task 

12. data intensive (no real time factor), ie. acquire and present 

» 

The following is the list of selected tasks/jobs for which an 
initial interview will be performed (dates indicate date of initial 

interview) : 

1. (diagnostics) medical diagnosis - HRL 

2. (diagnostics) equipment diagnosis - HRL 

3. (training) training pilots - HRL/Randolph 

4. (training) training a foreign language - HRL 

5. (high performance) flight controller console operations - Matt Barry, 

NASA/JSC - Dec. 11 

6. (high perforinance/KR) surgery - HRL/BAMC 

7. (form fill-out) - Bill Bayliss, SwRI Contracts - Oct. 26, 1990 

8. (people-oriented) personnel management - HLR/ Leadership Training 

9. (design) sofware design - Susan Crumrine, SwRI - ?? 

10. (planning) acquisition program management - Xavier Pena, Kelly AFB - 

Dec. 4, 1990 

11. (moni toring/ time) air traffic control - Jim Johnson, FAA Academy 

Nov. 1, 1990 

12. (perceptual) - HRL 

13. (bin-packing) cargo loading - HRL/Kelly 

14. (numerical) accounting - Linda Boehme, SwRI CPA - 77 

15. (data intensive/no time) - DRAIR Preparation - Gail Davis/00-ALC 

16. (planning) scientific protocol design - Herb Peel, SwRI - October 18, 

1990 
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January 10, 1991 


Intelligent Systems Evaluation 
Training Systems Division 
AFHRL/ID 
Brooks AFB, TX 78235 

Attention: Kurt Steuck, Ph.D. 

Subject: Monthly Progress Report from November 24 Through December 21, 1990; 

Research Activity No. ET.26; NASA Cooperative Agreement NCC9-16; 

Southwest Research Institute (SwRI) Project No. 05-3744 

Gentlemen: 

During this reporting period, we have continued to set-up and perform the initial interviews 
in the areas agreed upon. We have completed interviews in planning (acquisition program 
management), high performance (shuttle flight control), form fill-out, design (software design), a 
numerical task (accounting), a data intensive/no time constraint task (deficiency report analysis), 
and planning (scientific protocol design). These seven problem solving domains, along with the 
monitoring-with-time-constraint task (air traffic control) have been analyzed and values of 0-4 have 
been entered for each attribute on the task vs. characteristics matrix that we are maintaining (0 
meaning not used and 4 meaning essential). We plan to put this data into a spread sheet or 
database so that further analysis of similarities and differences between tasks can be made. 

We met with Capt. B. J. Jones on December 20, 1990 to discuss the potential for the Air 
Force to provide contacts in the remaining eight areas. We felt that their contacts would be better 
than any that we could find. We agreed to try to complete the initial round of interviews by late 
January or early February, depending on the accessibility of the experts. We also discussed with 
Capt. Jones what we thought we had learned so far from this process and our tentative plans for 
carrying-out the second set of interviews. The initial interview seems to take about 1 to 1.5 hours. 
We have found that the structure we have used in this initial interview has allowed us to get better 
information in less time than we have experienced in past developments. The questionnaire helps 
us to stay on track and the experts, so far, seem to really enjoy it. From this initial set of interviews 
we hope to develop some guidelines for how to proceed with the second interview. For example, 
we hope to generate rules such as "if the problem solving task has characteristics A, B, or C, then 
have the expert select some examples to work through" or "If the problem solving task has 
characteristics X and Y, then ask the following types of questions....". 
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During the next reporting period, we will continue to set-up and perform initial interviews. 
We plan to complete at least three of the remaining eight. The rest, five in all, will most likely take 
place in early February. 

Expenditures for this reporting period totaled S3, 854.92, including $118 in travel, leaving a 
balance of $48,136.40, including $150 of commitment for travel, of the original $65,909 contract 
amount. 


Approved: 

/ 

Richard B. Curtin 
Vice President 
Automation & Data Systems 
Division 

RBC/PKF/em 

Enclosure 

cc: Nancy Bell, RICIS 

Bill Bayliss, SwRI 
Tandy Herren, SwRI 


Sincerely, 



Pamela K. Fink, Ph.D. 
Manager 

Artificial Intelligence 
Section 


SCHEDULE FOR RESEARCH IN DEVELOPMENT 
OF A KNOWLEDGE ENGINEERING TAXONOMY 


Months from Date of Award 


1990 1991 


TASK | Aug Sep Oct Nov Dec Jan Feb Mar Apr May 


1 | C(10/18) 

2 | (2/28) D(5/10) 

3 | (3/30) 

4 | (4/26) 

5 | (3/30) 

6 | (4/26) 

7 j CCCCCDD D D(5/19) 


List of Tasks: 

Task 1 - Identification of a Set of Specific Problem Solving Tasks 

Task 2 - Definition of Identified Problem Solving Tasks 

Task 3 - Analysis of Identified Problem Solving Tasks 

Task 4 - Development of a Proposed Taxonomy of Problem Solving Tasks 

Task 5 - Analysis of Knowledge Engineering Techniques Used in Task 2 

Task 6 - Matching of Knowledge Engineering Techniques to Problem Solving Tasks 

Task 7 - Reporting 






